Posture Analysis Systems and Methods

ABSTRACT

Example posture analysis systems and methods are described. In one implementation, a computing system identifies a deviation of a user&#39;s posture from a predetermined posture based on visual data associated with the user. The user is informed of the deviation via a graphical image displayed on a display device. The user is then provided with instructions for correcting the deviation via the graphical image displayed on the display device.

RELATED APPLICATION

This application claims the priority benefit of U.S. ProvisionalApplication Ser. No. 62/362,391, entitled “Tracking and CorrectingPosture of a User,” filed on Jul. 14, 2016, the disclosure of which ishereby incorporated by reference herein in its entirety.

TECHNICAL FIELD

The present disclosure relates to systems and methods that analyze andtrack a user's posture.

BACKGROUND

In many situations, people spend significant amounts of time in front ofdesktop computers, laptop computers, display screens, or sitting inparticular chairs or other seating devices. Long hours in these types ofsituations may be associated with unhealthy posture habits, which theperson may be unaware of. In contrast, having a good posture may allow aperson to avoid certain problems associated with sitting, standing, orother positions with an incorrect posture.

Incorrect posture can cause problems a person's spine, back or neckwhich may result in pain or other problems. In some situations, healthmay be improved if proper posture is attained. There exists a need for asystem that helps a user maintain proper posture in a variety ofsituations.

BRIEF DESCRIPTION OF THE DRAWINGS

Non-limiting and non-exhaustive embodiments of the present disclosureare described with reference to the following figures, wherein likereference numerals refer to like parts throughout the various figuresunless otherwise specified.

FIG. 1A is a block diagram depicting an embodiment of a posture analysissystem.

FIG. 1B is a block diagram depicting an embodiment of a graphical imageas displayed by a posture analysis system.

FIG. 2 is a block diagram depicting an embodiment of a computing systemcapable of implementing a posture analysis system.

FIG. 3 is a block diagram depicting an embodiment of an image analysismodule.

FIG. 4 is a block diagram depicting an embodiment of a user feedbackmodule.

FIGS. 5A-5E represent a flow diagram depicting an embodiment of a methodto track and correct a user's posture in substantially real time.

FIG. 6 is a schematic diagram depicting an embodiment of a postureanalysis system interface that shows a registration of a predeterminedposture.

FIG. 7 is a schematic diagram depicting an embodiment of an interactiveposture correction sequence.

FIG. 8 is a schematic diagram depicting another embodiment of aninteractive posture correction sequence.

FIG. 9 is a schematic diagram depicting an embodiment of a sequence offrames to determine a user's posture.

FIG. 10 is a schematic diagram depicting an embodiment of a workflowthat illustrates the operation of a posture analysis system.

FIGS. 11A and B represent a flow diagram depicting an embodiment of amethod to track and correct a user's posture based on image recognition.

FIGS. 12A and B represent a flow diagram depicting an embodiment of amethod to characterize a user's posture.

FIG. 13 is a schematic diagram illustrating an embodiment of anapplication of a posture tracking system.

FIG. 14 is a schematic diagram illustrating an embodiment of anotherapplication of a posture tracking system.

DETAILED DESCRIPTION

In the following description, reference is made to the accompanyingdrawings that form a part thereof, and in which is shown by way ofillustration specific exemplary embodiments in which the disclosure maybe practiced. These embodiments are described in sufficient detail toenable those skilled in the art to practice the concepts disclosedherein, and it is to be understood that modifications to the variousdisclosed embodiments may be made, and other embodiments may beutilized, without departing from the scope of the present disclosure.The following detailed description is, therefore, not to be taken in alimiting sense.

Reference throughout this specification to “one embodiment,” “anembodiment,” “one example,” or “an example” means that a particularfeature, structure, or characteristic described in connection with theembodiment or example is included in at least one embodiment of thepresent disclosure. Thus, appearances of the phrases “in oneembodiment,” “in an embodiment,” “one example,” or “an example” invarious places throughout this specification are not necessarily allreferring to the same embodiment or example. Furthermore, the particularfeatures, structures, databases, or characteristics may be combined inany suitable combinations and/or sub-combinations in one or moreembodiments or examples. In addition, it should be appreciated that thefigures provided herewith are for explanation purposes to personsordinarily skilled in the art and that the drawings are not necessarilydrawn to scale.

Embodiments in accordance with the present disclosure may be embodied asan apparatus, method, or computer program product. Accordingly, thepresent disclosure may take the form of an entirely hardware-comprisedembodiment, an entirely software-comprised embodiment (includingfirmware, resident software, micro-code, etc.), or an embodimentcombining software and hardware aspects that may all generally bereferred to herein as a “circuit,” “module,” or “system.” Furthermore,embodiments of the present disclosure may take the form of a computerprogram product embodied in any tangible medium of expression havingcomputer-usable program code embodied in the medium.

Any combination of one or more computer-usable or computer-readablemedia may be utilized. For example, a computer-readable medium mayinclude one or more of a portable computer diskette, a hard disk, arandom access memory (RAM) device, a read-only memory (ROM) device, anerasable programmable read-only memory (EPROM or Flash memory) device, aportable compact disc read-only memory (CDROM), an optical storagedevice, and a magnetic storage device. Computer program code forcarrying out operations of the present disclosure may be written in anycombination of one or more programming languages. Such code may becompiled from source code to computer-readable assembly language ormachine code suitable for the device or computer on which the code willbe executed.

Embodiments may also be implemented in cloud computing environments. Inthis description and the following claims, “cloud computing” may bedefined as a model for enabling ubiquitous, convenient, on-demandnetwork access to a shared pool of configurable computing resources(e.g., networks, servers, storage, applications, and services) that canbe rapidly provisioned via virtualization and released with minimalmanagement effort or service provider interaction and then scaledaccordingly. A cloud model can be composed of various characteristics(e.g., on-demand self-service, broad network access, resource pooling,rapid elasticity, and measured service), service models (e.g., Softwareas a Service (“SaaS”), Platform as a Service (“PaaS”), andInfrastructure as a Service (“IaaS”)), and deployment models (e.g.,private cloud, community cloud, public cloud, and hybrid cloud).

The flow diagrams and block diagrams in the attached figures illustratethe architecture, functionality, and operation of possibleimplementations of systems, methods, and computer program productsaccording to various embodiments of the present disclosure. In thisregard, each block in the flow diagrams or block diagrams may representa module, segment, or portion of code, which includes one or moreexecutable instructions for implementing the specified logicalfunction(s). It will also be noted that each block of the block diagramsand/or flow diagrams, and combinations of blocks in the block diagramsand/or flow diagrams, may be implemented by special purposehardware-based systems that perform the specified functions or acts, orcombinations of special purpose hardware and computer instructions.These computer program instructions may also be stored in acomputer-readable medium that can direct a computer or otherprogrammable data processing apparatus to function in a particularmanner, such that the instructions stored in the computer-readablemedium produce an article of manufacture including instruction meanswhich implement the function/act specified in the flow diagram and/orblock diagram block or blocks.

The systems and methods described herein disclose a system that, usingvisual data, interactively monitors and tracks a user's posture insubstantially real time, and offers feedback to the user insubstantially real time if the user's posture deviates beyond a certainextent from a predetermined posture. This system can be implemented on acomputing system such as a laptop computer or a desktop computer, andincludes an imaging device such as a webcam or a video camera to acquirethe visual data. Feedback to the user may be displayed on, for example,a computer monitor or any other visual display device. Machine learningmethods and software libraries such as OpenCV may be used to implementcertain components of the posture analysis systems described herein.

FIG. 1A is a block diagram depicting an embodiment of a posture analysissystem 100. In some embodiments, posture analysis system includes acomputing system 102 that may be a laptop computer, a desktop computer,a mobile device, a tablet, or any other computing device or processingdevice capable of performing the functions and operations discussedherein. In some embodiments, computing system 102 may be remotelylocated, for example as a part of a cloud computing system. Computingsystem 102 is configured to receive visual data from an imaging device106 that may be a webcam, a video camera, a digital still image camera,or any other imaging device capable of capturing or identifying animage. Visual data received by computing system 102 from imaging device106 may be any combination of video data, still images, infrared imagesand so on. Computing system 102 is also configured to output visual datato a display device 108 that may be an LCD display, an OLED display, aCRT display, or any other display device. In some embodiments, imagingdevice 106 may be attached to display device 108 and may be physicallyoriented to capture visual data associated with a user 104. As usedherein, “visual data” includes data such as digital video, digital stillimages, infrared images, and the like. In some embodiments, the visualdata includes at least a portion of the user within an image frame fromany angle or perspective associated with imaging device 106. In otherembodiments, imaging device 106 may be integrated into display device108 (e.g. a webcam), such that user 104 is within the field of view ofimaging device 106 when looking at display device 108. In other words,imaging device 106 is configured to capture an image of at least aportion of user 104 while user 104 performs one or more specific taskson computing system 102. In some embodiments, user 104 may positionthemselves such that the optical axis of imaging device 106 issubstantially orthogonal to the left pectoral muscle and right pectoralmuscle of user 104.

In some embodiments, the operation of posture analysis system 100consists of two steps. A first step is an initialization step where user104 assumes a predetermined posture that is captured by imaging device106 and stored by computing system 102. In some embodiments, the term“predetermined posture” may be defined as a posture used to initializeposture analysis system 100 and used as a basis for tracking a currentposture of the user. In some embodiments, the term “predeterminedposture” may be also referred to as a “reference posture,” a “preferredposture,” a “healthy posture,” a “good posture,” and so on. For example,the predetermined posture may be an ideal or preferred posture for theparticular user. In particular embodiments, image processing techniquessuch as computer vision, facial detection and facial recognition may beused in conjunction with machine learning algorithms to analyze andappropriately characterize the visual data received by computing system102. In some embodiments, computer vision software libraries, such asOpenCV, may be used to implement certain functions of posture analysissystem 100, such as analysis of visual data.

In some embodiments, a second step associated with the operation ofposture analysis system 100 is a tracking step where a current postureof user 104 is tracked by computing system 102 in substantially realtime based on visual data received from imaging device 106. In someembodiments, the term “substantially real time” includes operation witha small time delay such as a few seconds or a fraction of a second. Inother embodiments, substantially real time may include operation with alonger time delay up to several minutes. In particular embodiments, atemporal history associated with the tracking of the posture of user 104may be stored and presented to the user at a later time (for example, ondisplay device 108 or via email).

In some embodiments, the current posture of user 104 is tracked relativeto the predetermined posture. In some situations, if the tracked currentposture of user 104 deviates from the predetermined posture by more thana certain threshold, computing system 102 may inform user 104 about thedeviation via a graphical image 110 displayed on display device 108. Insome embodiments, user 104 may be informed about the deviation viagraphical image 110 in substantially real time. In particularembodiments, graphical image 110 may simultaneously include an imagedepicting the predetermined posture, an image depicting the currentposture of user 104, and information that may allow user 104 tosubstantially eliminate the deviation and return to the predeterminedposture. In some embodiments, information presented in graphical image110 may be any combination of graphical symbols, textual information anda video rendering of user 104. If user 104 follows the presentedinformation to eliminate the deviation and assumes the predeterminedposture again (i.e., if user 104 corrects their posture), computingsystem 102 may detect that user 104 has assumed the predeterminedposture based on the visual data, and may then remove graphical image110 from display device 108. Details of the initialization and trackingsteps are discussed in greater detail herein.

In some embodiments, computing system 102 may associate the currentposture of user 104 with a specific task being performed by user 104. Inparticular embodiments, computing system 102 may associate a deviationof the posture of user 104 from the predetermined posture with aspecific task being performed by user 104. In some embodiments, the taskbeing performed by user 104 is an activity being performed by user 104.Examples of the task being performed by the user include using anapplication executing on computing system 102, reading a book, writing,riding in a vehicle, performing surgery in an operating room, practicingyoga, or any other kind of activity. Posture analysis system 100 may beused in a variety of situations such as operating rooms, airplanes (toensure a passenger assumes a healthy posture), cars, and so on.

FIG. 1B is a block diagram depicting an embodiment of a graphical image110 as displayed by a posture analysis system 100. In some embodiments,graphical image 110 may include a line drawing of a predeterminedposture 112, and a graphic element 114 corresponding to the currentposture of user 104. In some embodiments, graphic element 114 may be anellipse that corresponds to a position of the face of user 104 in aframe associated with the visual data as determined by facial detectionalgorithms running on computing system 102. In some embodiments, graphicelement 114 may be a polygon. In particular embodiments, graphic element114 may be any combination of a cartoon character, a photographicdepiction of a head, shoulders and a body, a three-dimensional meshrendering of a head, shoulders and a body, a cut-out rendition of theface of user 104 on a comical background, an image of another person, ananimal (e.g., a bunny), and the like. In some embodiments, graphicalimage 110 may be generated by computing system 102 and rendered ondisplay device 108 in substantially real time.

The position of graphic element 114 in graphical image 110 correlateswith a spatial position of the face of user 104 which, in turn, can bemapped to the current posture associated with user 104. In someembodiments, graphical image 110 may also include a text box 116 thatprovides posture correction information to user 104. If user 104 followsthe posture correction information, user 104 can be guided back toassume the predetermined posture. FIG. 1B depicts text box 116 asinforming user 104 to move right and back (i.e., away from displaydevice 108) to achieve the predetermined posture. Further details of theposture correction step that involve substantially eliminating adeviation associated with a user's current posture from thepredetermined posture are provided herein.

FIG. 2 is a block diagram depicting an embodiment of computing system102 capable of implementing posture analysis system 100. In someembodiments, computing system 102 may include an image analysis module118 that is configured to process visual data received from imagingdevice 106. In some embodiments, image analysis module 118 may implementat least one software algorithm such as facial recognition, facialdetection, machine learning, computer vision, and so on when analyzingthe visual data. In particular embodiments, image analysis module 118may use software libraries for computer vision, such as OpenCV. Imageanalysis module 118 may be configured to detect a user's face and otherparts of the user's body, such as the user's shoulders based on visualdata received from imaging device 106. In some embodiments, imageanalysis module 118 may be configured to compare a current user posturewith a predetermined posture and compute a deviation associated with thecurrent user posture from the predetermined posture. Further details ofimage analysis module 118 are provided herein.

In some embodiments, computing system 102 includes one or moreprocessors 120 that are configured to perform processing functions thatinclude, but are not limited to, mathematical and arithmeticcomputations or any other computational functions. One or moreprocessors 120 may be any combination of microprocessors,microcontrollers, digital signal processors (DSPs), field-programmablegate arrays (FPGAs), or any other processing devices. Computing system102 may also include one or more memory devices 122 that are configuredto store data in any combination of volatile or non-volatile formats.Examples of memory devices 122 include random access memory (RAM),read-only memory (ROM), NAND flash memory, and so on.

In some embodiments, computing system 102 may include a user feedbackmodule 124 that is configured to generate feedback associated with auser's posture to the user when a deviation in the user's posturerelative to the predetermined posture is above a certain threshold.Details of user feedback module 124 are provided herein. In someembodiments, computing system 102 may include one or more mass storagedevices 126, that include any combination of magnetic hard disk drives,solid-state drives, or any other type of storage mechanism. Mass storagedevices 126 may be configured to store data in a non-volatile format foraccess at a later time.

One or more input/output devices 128 may also be included in computingsystem 102. In some embodiments, input/output devices 128 may includeany combination of keyboards, computer mice, computer video terminals orscreens, audio input/output devices (microphones and speakers),touchscreens, or any other device that allows a user to interact withcomputing system 102.

Computing system 102 may also include one or more interfaces 130 thatare configured to allow coupling between processing system 102 and otherexternal devices. Interfaces 130 may include, for example, one or moreuniversal serial bus (USB) ports, IEEE 1394 (Firewire) ports, IEEE802.11 (WiFi) interfaces, Bluetooth interfaces, and the like.

In some embodiments, computing system 102 may include a communicationmodule 132 that is configured to communicate, for example, with imagingdevice 106, display device 108, and any other components, systems,devices, or routines. In some embodiments, communication module 132 maybe responsible for implementing different communication protocolsassociated with different devices.

In some embodiments, the different subcomponents of computing system 102as described herein may each be coupled to a central data bus 134 thatis configured to transmit communication signals and data between thedifferent subcomponents of computing system 102.

FIG. 3 is a block diagram depicting an embodiment of image analysismodule 118. In some embodiments, image analysis module 118 is configuredto perform analysis on visual data acquired from imaging device 106. Toachieve this, image analysis module 118 may include different componentsthat perform different functions. For example, image analysis module 118may include a background initialization manager 302 that is configuredto compensate for any background elements present in the visual data. Insome embodiments, a user may not be sitting in front of a plainbackground. For example, the background associated with the user and thefield of view of imaging device 106 may not be static. Instead, thebackground may contain dynamic visual information associated with, forexample, moving objects in the background. In some embodiments,background initialization manager 302 performs functions to characterizethe background associated with the user. In some embodiments, thischaracterization may be performed in the user's presence. In otherembodiments, this characterization may be performed in the user'sabsence. Background initialization manager 302 may include one or morevisual reference points from the background as a part of any backgroundinitialization data that may be used for further processing. In someembodiments, characterizing the background associated with the user isperformed by asking the user to sway from side to side, capturing theswaying motion of the user, and then using, for example, optical flowalgorithms to determine any portions of the associated captured visualdata that are non-stationary with respect to time. Non-stationary areasof the visual data are associated with the user swaying motion, versusthe background not moving and associated with temporally static visualdata. The temporally static visual data is used to characterize orinitialize the background. In some embodiments, the user may be asked tosway to their left, sway to their right, and then return back to theircenter to complete the background characterization (initialization)process.

In some embodiments, image analysis module 118 may include a backgroundtracking manager 304 that is configured to perform dynamic tracking onthe background associated with the user in order to compensate for anydynamic components that may affect posture tracking of the user. Forexample, in the event that facial detection algorithms are included inthe posture tracking process implemented by computing system 102, theremay be other people with faces in the background. In some embodiments, aperson may be looking over the shoulder of the user, for example, inresponse to the user wanting to share some information with the person.In such cases, the image recognition algorithm may recognize more thanone face in the visual data and this might lead to errors in the posturetracking process. Background tracking manager 304 is configured toaccount for such errors and to prevent them from occurring. For example,a face in the background will be at a greater distance from imagingdevice 106 as compared to the face of user 104, as a result of which thelatter will be recognized as a larger ellipse or blob by a facialdetection algorithm as compared to the former. Background trackingmanager is configured to track the largest ellipse or blob in a frameassociated with the visual data as that is most likely the user's facewhile eliminating any other potentially distracting elements.

In some embodiments, background tracking manager 304 may be configuredto compensate for any inadvertent movement in imaging device 106relative to the position of user 104. For example, if user 104 is usinga laptop and changes the tilt or orientation of the screen, thebackground data associated with user 104 may change. Similarly, thebackground data associated with user 104 may change if user 104 changesa position of the laptop (e.g., user 104 rotates the laptop, or movesthe laptop towards or away from themselves). Background tracking manager304 may be configured to account for such changes in the background bystoring any key visual characteristics of the background andsubsequently compensating accordingly. In some embodiments, after aninitialization of the system is completed, user 104 is not required tore-initialize a predetermined posture.

Image analysis module 118 may also include a user initialization module306 that is configured to initialize the system and store apredetermined posture using techniques described herein. In someembodiments, image analysis module 118 may include a posture detectionmanager 308 that may use image processing algorithms such as facialrecognition or face detection to detect the current posture of user 104.In particular embodiments, image analysis module 118 may include aposture tracking manager 310 that is configured to track the currentposture of user 104 in real time or substantially real time relative toa predetermined posture. In some embodiments, posture tracking manager310 may use image processing algorithms such as facial recognition orface detection to track the current posture of user 104.

In some embodiments, image analysis module 118 may include apredetermined posture generator 312 that is configured to retrievestored data associated with the predetermined posture and generate arendition of the predetermined posture to be compared with the currentposture of user 104. In some embodiments, image analysis module 118 mayinclude a posture comparator 314 that compares the predetermined posturewith the current posture of user 104 to compute any deviation in thecurrent posture of user 104 from the predetermined posture. In someembodiments, a deviation in the current posture of user 104 from thepredetermined posture is calculated by computing a pixel distance in thevisual data between the predetermined posture and the current posture ofuser 104. In particular embodiments, the pixel distance—up, down, to theleft, to the right, or any combination thereof—is a measure of thephysical distance or physical deviation of the current posture of user104 from the predetermined posture. In some embodiments, a percentagedifference (increase or decrease) is used to determine any deviation inthe current posture of user 104 from the predetermined posture in adirection towards or away from imaging device 106 respectively. If user104 moves towards imaging device 106, the apparent size of user 104 asrendered by imaging device 106 increases relative to the predeterminedposture, and vice versa. In some embodiments, the face of user 104 asdetected using facial detection may be characterized as an ellipticalshape in the visual data. The length and width (major and minor axesrespectively) of this elliptical shape may be compared to a similarelliptical shape corresponding to the user's face associated with thepredetermined posture. A relative increase or decrease in the number ofpixels (for example, the area) of the elliptical shape corresponding tothe face of user 104 relative to the elliptical shape corresponding tothe predetermined posture may be used to determine whether user 104 hasdeviated from the predetermined posture while leaning towards or awayfrom imaging device 106 respectively. If the deviation exceeds a certainpredetermined threshold, an alert generator 316, included in imageanalysis module 118, is configured to generate one or more alerts thatmay include visual alerts, audible alerts, textual alerts, and so on.

In some embodiments, the predetermined threshold may be viewed as asystem design parameter, and can be tuned to suit specific applications.For example, the predetermined threshold may be viewed as the number ofpixels in the visual data by which the current posture of user 104 needsto deviate from the predetermined posture before the user is alertedabout the deviation. In some embodiments, a relatively small thresholdsuggests a sensitive system that alerts user 104 for relatively smalldeviations in the current posture of user 104 from the predeterminedposture. On the other hand, a relatively large threshold suggests a lesssensitive or relatively relaxed system that responds only to largerdeviations in the current posture of user 104 from the predeterminedposture. In some embodiments, the predetermined threshold may be presetinto posture analysis system 100. In particular embodiments, thepredetermined threshold may be user-customizable.

In some embodiments, image recognition techniques may be implementedthat preform recognition on a user's face to characterize the user'seyes, nose and lips to provide enhanced characterization of the currentposture of user 104. Such techniques are especially useful in caseswhere, for example, the user might have rotated their head or if imagingdevice 106 is positioned to the side of user 104 or far below or farabove user 104.

FIG. 4 is a block diagram depicting an embodiment of user feedbackmodule 124 that is configured to provide feedback to a user of postureanalysis system 100. In some embodiments, user feedback module 124generates feedback for the user if the deviation associated with theuser's current posture from the predetermined posture exceeds a certainthreshold. User feedback module 124 may include a text feedbackgenerator 402 that is configured to generate text messages that may beincluded, for example, in graphical image 110.

In some embodiments, user feedback module 124 may include an audiofeedback generator 404 that is configured to generate audio signals thatmay be played back on, for example, a speaker associated with computingsystem 102 or display device 108 (not shown). In some embodiments, userfeedback module 124 may include a visual feedback generator 406 that isconfigured to generate visual feedback signals to user 104 that informuser 104 about any deviation in the current posture of user 104 from thepredetermined posture. In some embodiments, the visual feedback signalsmay be provided to user 104 via graphical image 110.

FIG. 5A represents a flow diagram depicting an embodiment of a method500 to track and correct a user's posture in substantially real time. At502, the method receives a command from a computing system (such ascomputing system 102) to begin tracking a user's posture. At 504, themethod determines whether the computing system is initialized with apredetermined posture. If the computing system is initialized with apredetermined posture the method goes to A, with a continued descriptionprovided herein. If, at 504, the computing system is not initializedwith a predetermined posture, the method goes to 506, where the user isprompted to assume the predetermined posture. In some embodiments, theuser may be prompted to assume the predetermined posture by computingsystem 102 via any combination of text and graphics rendered ongraphical image 110 as displayed on display device 108. In someembodiments, the user may be assisted in assuming the predeterminedposture by offering, for example, text prompts to the user on graphicelement 114 such as “sit up straight,” or “make sure your shoulders arenot slouching,” and so on. At 508, the method receives reference visualdata associated with the user, where the reference visual data isassociated with the user assuming the predetermined posture. In someembodiments, the reference visual data may be received from imagingdevice 106. At 510, the method determines the user's posture along afirst axis based on a size of the user's face using, for example, facialdetection on the reference visual data. In some embodiments, the firstaxis is substantially parallel to the optical axis of imaging device 106which, in turn, may be substantially orthogonal to the plane of displaydevice 108. In particular embodiments, a user may move their body alongthe first axis. The motion of the user along the first axis causes thesize of the user's face in the visual data to change. When the usermoves closer to imaging device 106 the user's face appears larger in thevisual data and vice versa. When the user assumes the predeterminedposture, the size of the user's face is associated with thepredetermined posture that corresponds to a specific distance fromimaging device 106 along the optical axis of imaging device 106; this isthe basis used for step 510.

At step 512, the method determines the user's posture along a secondaxis based on a first position of the user's face along the second axisusing, for example, facial detection on the reference visual data. Insome embodiments, this second axis may be substantially orthogonal tothe optical axis of imaging device 106 and substantially parallel to aplane associated with display device 108. At step 514, the methoddetermines the user's posture along a third axis based on a firstposition of the user's face along the third axis using, for example,facial detection on the reference visual data. In some embodiments, thisthird axis may be substantially orthogonal to the optical axis ofimaging device 106, substantially parallel to the plane associated withdisplay device 108, and substantially orthogonal to the second axis.Steps 512 and 514 aim to characterize the position of the user's face intwo dimensions in a plane substantially parallel to the plane associatedwith display device 108. In some embodiments, the combination of thefirst axis, the second axis and the third axis forms athree-dimensional, substantially orthogonal coordinate system. Steps510, 512 and 514 together may be used to provide a three-dimensionalspatial characterization (using the first axis, second axis, and thirdaxis) of the position of the user's face, and hence the user's posture,based on the three-dimensional, substantially orthogonal coordinatesystem. This feature allows posture analysis system 100 to performthree-dimensional tracking of a current posture associated with theuser. In some embodiments, the motion of the user's face along each ofthe three axes may be any combination of translation and rotationalmotion. The method then goes to B, with a continued description providedsubsequently.

FIG. 5B is a continued description of the method 500. Starting at B, themethod 500 goes on to include spatial data associated with the user'sshoulders to augment the three-dimensional characterization of theuser's posture based on facial detection steps 510 through 514. At 516,the method determines a position of the user's left shoulder using thereference visual data. At 518, the method determines a position of theuser's left shoulder using the reference visual data. In someembodiments, steps 516 and 518 may be accomplished by the user swayingfrom side to side in front of imaging device 106 so that postureanalysis system 100 can determine the position of the user's leftshoulder and the position of the user's right shoulder from the visualdata. In other embodiments, steps 516 and 518 may be accomplished byhaving the user wiggle their shoulders in front of imaging device 106.

In other embodiments, steps 516 and 518 may be accomplished by havingthe user stroke their left shoulder and right shoulder respectively,with image detection techniques being used to determine a position ofthe user's left shoulder and a position of the user's right shoulder. Inparticular embodiments, an area below the detected face of the user maybe divided into a left portion and a right portion, with a search beingperformed in each of the left portion and the right portion to detectthe user's left shoulder and right shoulder respectively.

At 520, the method determines the user's posture in three dimensionsbased on the user's posture along the first axis, the user's posturealong the second axis, the user's posture along the third axis, theposition of the user's left shoulder and the position of the user'sright shoulder. In some embodiments, data associated with modeling theuser's posture in three dimensions includes using an ellipse or polygonas an output of the facial detection system that is used to model theuser's face. The centroid of the ellipse or polygon is determined, andthe coordinates of the centroid of the ellipse or polygon relative to atwo-dimensional coordinate system comprising the second axis and thethird axis is stored along with the length, width and area of theellipse or polygon. The length, width and area of the ellipse or polygongive a measure of the motion of the user relative to the first opticalaxis. The measurements of the coordinates of the centroid of the ellipseor polygon and the length, width and area of the ellipse or polygonprovide a three-dimensional characterization or modeling of the posturedata.

In some embodiments, at 520, the upper left location of each shoulderrelative to a coordinate system formed by the second axis and the thirdaxis is used to model posture data associated with the user's shoulders.Next, at 522, the method stores the user's posture as a predeterminedposture. The method continues to A, with a continued descriptionprovided subsequently.

FIG. 5C is a continued description of the method 500. While steps 506through 522 described the generation of the predetermined posture (thatcan also be referred to as system calibration), subsequent stepsdescribe the operation of the system with respect to posture monitoringand tracking for a given user associated with a predetermined posture.

Starting at A, the method continues to 524, where the computing systemreceives current visual data associated with the user. “Current visualdata” is used to refer to visual data associated with user posturemonitoring and tracking. In some embodiments, the current visual datamay be received from imaging device 106. At 526, the method determinesthe user's current posture along the first axis based on a size of theuser's face using facial detection on the current visual data. Thisprocess is similar to step 510. At 528, the method determines the user'scurrent posture along the second axis based on a first position of theuser's face along the second axis using facial detection on the currentvisual data. This process is similar to step 512. At 530, the methoddetermines the user's current posture along the third axis based on asecond position of the user's face along the third axis using facialdetection on the current visual data. This process is similar to step514. Next, at 532, the method determines a current position of theuser's left shoulder using the current visual data. This process issimilar to step 516. At 534, the method determines a current position ofthe user's right shoulder using the current visual data. This process issimilar to step 518. The method then proceeds to C, with a continueddescription provided subsequently.

FIG. 5D is a continued description of the method 500. At 536, the methoddetermines the user's current posture in three dimensions based on theuser's current posture along the first axis, the user's current posturealong the second axis, the user's current posture along the third axis,the current position of the user's left shoulder and the currentposition of the user's right shoulder. This process is similar to step520. Next, at 538, the method tracks the user's current posture insubstantially real time. In some embodiments, tracking the user'scurrent posture is performed at a rate that is similar to the frame rateof imaging device 106, for example, 30 frames per second. Otherembodiments may implement faster or slower tracking rates depending onthe specific system or user requirements or available system resources.For example, frame rates may range from 5 frames per second to 2000frames per second. At 540, the method associates the user's currentposture with a specific task being performed by the user (e.g., a taskbeing performed on the computing system). This association of the user'scurrent posture with a particular task may allow posture analysis system100 to potentially identify bad posture habits associated withperforming the particular task. For example, a user may slouch whileusing a word processor, crane their neck while reading an email, orshrug their shoulders when watching a video. All these associations ofbad posture habits can be associated with a specific task beingperformed by the user and then provided to the user at a later time toallow them to take corrective steps for assuming a healthy posture.

At 542, the method compares the user's current posture with thepredetermined posture. At 544, the method determines whether the user'scurrent posture has deviated from the predetermined posture. In someembodiments, steps 542 and 544 may be accomplished by posture comparator314. If the user's posture has not deviated from the predeterminedposture the method returns to A. If the user's posture has deviated fromthe predetermined posture the method continues to D, with a continueddescription provided subsequently.

FIG. 5E is a continued description the method 500. At 546, the methodassociates the deviation with the specific task being performed by theuser. While step 540 associates the user's current posture with aspecific task being performed by the user on the computing system, step546 associates the deviation associated with the user's current posturefrom the predetermined posture with the specific task being performed bythe user. This allows posture analysis system 100 to characterize badposture habits associated with a user in relation to a specific taskbeing performed by the user. Next, at 548, the method determines whetherthe deviation is greater than a predetermined threshold. In someembodiments, each of the first axis, the second axis, and the third axisis monitored to determine a deviation in the user's current posture fromthe predetermined posture. A deviation of the user's current posturefrom the predetermined posture along one or more axis by over athreshold associated with that axis is sufficient for method 500 totrigger an alert or suggestion to the user. In one example, thethresholds associated with the axes are:

Approximately 32 pixels along the second axis in either direction, withthe second axis being substantially parallel to a horizon associatedwith the user.

Approximately 18 pixels along the third axis in either direction, withthe third axis being substantially orthogonal to the second axis.

An approximately 10% to 30% deviation in either direction in the size(length, width and area) of an ellipse or polygon associated with theuser's face along the third axis, with the third axis beingsubstantially orthogonal to the plane formed by the second axis and thethird axis.

In some embodiments, an alert may be generated for a deviation of theuser along the second axis by 32 pixels or more. However a correctivemotion of approximately 15 pixels and the user assuming the associatedposture for a predetermined amount of time may cause posture detectionsystem 100 to remove the alert.

If the deviation is greater than the predetermined threshold then themethod continues to 550, where the method informs the user insubstantially real time of the deviation via a graphical image (forexample, graphical image 110) on a display device (for example, displaydevice 108). The graphical image includes an image depicting thepredetermined posture and a simultaneous representation of the user'scurrent posture. In some embodiments, the predetermined posture may bedepicted using a line rendition. In particular embodiments, the user'scurrent posture may be represented by an ellipse corresponding to thecurrent position of the user's face as determined by facial detectionalgorithms running on the computing system. In some embodiments, therepresentation of the predetermined posture or the user's currentposture may be customizable by the user. Details about the presentationand operation of the graphical image are presented herein.

In some embodiments, method 500 may also include any combination ofaveraging, smoothing, or probabilistic functions that prevent the systemfrom triggering false alarms due to any measurement errors associatedwith posture analysis system 100. For example, a measurement error thatshows a user posture “jumping” by too many pixels may be filtered (orignored) by these functions as they are associated with false alarms andmeasurement uncertainties rather than any realistic motion by the user.As used herein, the term “filtering” includes system operations such asavoiding pixels or other items, ignoring pixels or other items,accounting for pixels or other items, rejecting pixels or other items,disregarding pixels or other items, and so on. In some embodiments, thegoal of the filtering operation is to avoid taking into considerationunrealistic jumps in the user's current posture as determined from thevisual data due to measurement errors associated with the system.Including the filtering operation as a part of the functionality ofposture tracking system 100 enhances the operation and accuracy ofposture analysis system 100 by, for example, eliminating or reducing theprobability of any occurrences of false alarms or false positioningassociated with deviations in the current user's posture.

In some embodiments, posture analysis system 100 may introduce a timedelay between step 548 and step 550. In particular embodiments, the timedelay may be approximately ⅓ seconds, or 320-370 milliseconds.Introducing the time delay is done to account for a user swaying backand forth about a mean position that is the predetermined posture, aprocess considered acceptable for healthy posture. Without the built-inlag, posture analysis system 100 may exhibit an overly sensitiveresponse to user inputs. In alternate embodiments, any value may be usedas a time lag in posture analysis system 100.

At 552, the method presents a suggested directional change in posture tothe user to reduce the deviation. Essentially, this step is a posturecorrection suggestion step. In some embodiments, suggestions may beprovided by any combination of text messages included graphical symbolssuch as arrows, or any other similar symbols included in, for example,graphical image 110. Step 552 may also include audio feedback to theuser in the form of coded messages (e.g., beeps) or explicit verbalfeedback (via, for example, prerecorded voice messages). The method thengoes back to A.

Returning to 548, if the method determines that the deviation is notgreater than the predetermined threshold then the method goes to 554,where the graphical image displayed on the displayed device, if present,is removed, and the method returns back to A.

FIG. 6 is a schematic diagram depicting an embodiment of a postureanalysis system interface 600 that shows a registration of apredetermined posture. In some embodiments, the process of postureregistration is performed using a graphical image 602 displayed on, forexample, imaging device 108. In some embodiments, graphical image 602may be identical to graphical image 110.

In some embodiments, when the user assumes a predetermined posture afterbeing prompted (for example, in step 506 of method 500), the user maypress a certain key on the keyboard or click a graphical buttondisplayed on display device 108 using their mouse or use a predeterminedor pre-established gesture that may be captured by imaging device 106and recognized by a computer vision system to confirm that they haveassumed the predetermined posture. In response to the confirmation,computing system 102 receives visual data, processes the visual data toextract the predetermined posture, and stores the predetermined posture.In some embodiments, the user may be given a time frame (e.g., 5 secondsor 10 seconds) to assume and maintain the predetermined posture. Duringthis time frame, the computing system receives visual data, processesthe visual data to extract the predetermined posture, and stores thepredetermined posture.

In some embodiments, computing system 102 may render the predeterminedposture as a line drawing 604 on graphical image 602, along with anellipse 606 that provides a location of the user's face, where ellipse606 is substantially congruent to and substantially identically locatedwith the face portion of line drawing 604. In some embodiments, ellipse606 may be replaced by other graphic symbols that may be userselectable, such as cartoon characters, caricatures, renditions ofanimals, flowers, other kinds of avatars, and the like. In someembodiments, upon successful registration of the predetermined posture,computing system 102 may present a text message 608 to the userconfirming that the posture has been registered. For example, asdepicted in FIG. 6, text message 608 may read “Posture Registered.” Theposture analysis system 100 is now configured to begin tracking theuser's posture in substantially real time and informing the user of anydeviations from the predetermined posture in substantially real time.

FIG. 7 is a schematic diagram depicting an embodiment of an interactiveposture correction sequence 700. In some embodiments, a user's currentposture may deviate from the predetermined posture. If the computingsystem determines that this deviation is greater than a predeterminedthreshold, computing system 102 may alert user 104 by presentingrelevant information on a graphical image 702 that includes a linerendition 714 of the predetermined posture. In some embodiments,graphical image 702 may also depict an ellipse 712 corresponding to athree-dimensional position of the user's face relative to the referenceposture as represented by line rendition 714. In this case, thedeviation of the user's current posture from the predetermined postureis shown to be to the left of line rendition 714. Graphical image 702may also include textual information such as a text message 710 thatprompts the user to shift their posture in a particular direction toreduce or eliminate the deviation. For example, FIG. 7 shows textmessage 710 displaying the prompt “shift to your right,” to allow theuser to compensate for the deviation to the left. In addition to textmessage 710, audio feedback via encoded audio signals or explicit voicerecordings may be provided to the user.

FIG. 7 depicts a sequence of events that occurs as displayed to a user,starting with graphical image 702, as the user corrects their posture.In response to the prompt on text message 710, as the user starts toshift to their right, correction sequence 700 may replace graphicalimage 702 by a graphical image 704 that includes line rendition 714. Asthe user responds to text message 710 and moves to their right, therendition of the user's face also moves to the right, as depicted byellipse 712. In graphical image 704, ellipse 712 is shifted to the rightas compared to the position of ellipse 712 in graphical image 702,corresponding to the user's response to the prompt on text message 710.Since there still is a deviation in the user's posture compared to thepredetermined posture text message 710 in graphical image 704 stillincludes text message 710 “shift to your right.” In addition to textmessage 710, audio feedback via encoded audio signals or explicit voicerecordings may be provided to the user.

As the user moves further to their right in response to the prompt ontext message 710, correction sequence 700 replaces graphical image 704by a graphical image 706 that includes line rendition 714, as ingraphical image 702 and graphical image 704. As the user responds totext message 710 and moves to their right, the rendition of the user'sface also moves to the right, as depicted by ellipse 712. In graphicalimage 706, ellipse 712 is shifted to the right as compared to theposition of ellipse 712 in graphical image 704, corresponding to theuser's response to the prompt on text message 710. While the user iscloser to the predetermined posture in graphical image 706 as comparedto graphical image 704, there still is a deviation in the user's posturecompared to the predetermined posture; hence text message 710 ingraphical image 706 still reads “shift to your right.” In addition totext message 710, audio feedback via encoded audio signals or explicitvoice recordings may be provided to the user.

Once the user assumes the predetermined posture as shown by a graphicalimage 708, a rendition of the user's face via ellipse 712 now coincideswith the representation of the user's face on line rendition 714 thatdepicts the predetermined posture, as depicted on graphical image 708.Since the user has now assumed the predetermined posture and correctedfor the deviation, text message 710 in graphical image 708 may nowdepict a check mark 716 that provides a confirmation to the user thatthey have assumed the predetermined posture. In some embodiments, checkmark 716 may be replaced with a congratulatory message or some othercongratulatory graphic that may be displayed while posture analysissystem may also provide audio feedback to the user via encoded audiosignals or explicit voice recordings.

In some embodiments, posture analysis system 100 may remove thegraphical image, such as graphical image 708, from display device 108after a certain time interval once the user assumes and maintains thepredetermined posture. In some embodiments, processing system 102 mayprocess visual data at a lower frame rate once the user has assumed thepredetermined posture with graphical image 708 removed from displaydevice 108. This is done to reduce any computing resource utilization byposture analysis system 100 on computing system 102 when the usermaintains the predetermined posture. When the user deviates from thepredetermined posture and needs interactive feedback, processing system102 may process visual data at a higher frame rate to providesubstantially real time updates to the user via a graphical image suchas graphical image 702.

FIG. 8 is a schematic diagram depicting another embodiment of aninteractive posture correction sequence 800. In some embodiments, auser's current posture may deviate from the predetermined posture. Ifthe computing system determines that this deviation is greater than apredetermined threshold, computing system 102 may alert user 104 bypresenting relevant information on a graphical image 802 that includes aline rendition 814 of the predetermined posture. In some embodiments,graphical image 802 may also depict an ellipse 812 corresponding to athree-dimensional position of the user's face relative to the referenceposture. In this case, the deviation in the user's current posture fromthe predetermined posture is in a direction towards imaging device 106(i.e., the user's current position is too close to imaging device 106relative to the predetermined posture), resulting in larger facialdimensions as determined by a facial detection algorithm running oncomputing system 102. Graphical image 802 may also include textualinformation such as a text message 810 that prompts the user to shifttheir posture in a particular direction to reduce or eliminate thedeviation. For example, FIG. 8 shows text message 810 displaying theprompt “move backwards.” In addition to text message 810, audio feedbackvia encoded audio signals or explicit voice recordings may be providedto the user.

FIG. 8 depicts a sequence of events that occurs as displayed to a userstarting with graphical image 802 as the user corrects their posture. Inresponse to the prompt on text message 810, as the user starts to movebackwards, correction sequence 800 may replace graphical image 802 witha graphical image 804 that includes line rendition 814. As the userresponds to text message 810 and moves backwards, the size of ellipse812 in graphical image 814 associated with the rendition of the user'sface reduces and beings to approach the size of the face in thepredetermined posture as represented by line rendition 814. (The size ofthe user's face is depicted by ellipse 812.) Since there still is adeviation in the user's posture compared to the predetermined posture,text message 810 in graphical image 804 still reads “move backwards.” Inaddition to text message 810, audio feedback via encoded audio signalsor explicit voice recordings may be provided to the user.

As the user continues to move backwards in response to the prompt ontext message 810, correction sequence 800 replaces graphical image 804by a graphical image 806 that includes line rendition 814, as ingraphical image 802 and graphical image 804. As the user responds totext message 810 and moves backwards, the rendition of the user's face(i.e., the size of ellipse 812 used to depict the user's face ingraphical image 806) also reduces in size. While the user is closer tothe predetermined posture, there still is a deviation in the user'sposture compared to the predetermined posture; hence text message 810 ingraphical image 806 still reads “move backwards.” In addition to textmessage 810, audio feedback via encoded audio signals or explicit voicerecordings may be provided to the user.

Once the user assumes the predetermined posture to reach a final state808, a rendition of the user's face via ellipse 812 now coincides withthe representation of the user's face on line rendition 814 that depictsthe predetermined posture, as depicted on graphical image 808. Since theuser has now assumed the predetermined posture and corrected for thedeviation, text message 810 in graphical image 808 may now depict acheck mark 816 that provides a confirmation to the user that they haveassumed the predetermined posture. In some embodiments, check mark 816may be replaced with a congratulatory message or some othercongratulatory graphic that may be displayed while posture analysissystem may also provide audio feedback to the user via encoded audiosignals or explicit voice recordings.

In some embodiments, feedback provided to the user to correct theirposture may include shading at least one portion of graphical image 110based on the deviation of the user's current posture from thepredetermined posture. In particular embodiments, the term “shading” isused to signify operations such as reducing the luminosity of at leastone portion of graphical image 110, changing the color of at least oneportion of graphical image 110, and the like. In some embodiments, theshading may be graduated to transition (or fade) into the remainingportion of graphical image 110 that is not shaded. In other embodiments,there may be a high contrast between a shaded portion and an unshadedportion of graphical image 110. In particular embodiments, the shadingmay be transparent. In other embodiments, the shading may be opaque. Instill other embodiments, the shading may be translucent.

In some embodiments, a portion of graphical image in a directioncorresponding to the deviation may be shaded. For example, if the userdeviates to their right, a right-hand portion of graphical image 110 maybe shaded. If the user deviates above the predetermined posture, a topportion of graphical image 110 may be shaded. Forwards or backwardsdeviations may be denoted by introducing vignetting (i.e., lightening ordarkening a periphery) of graphical display 110. Combinations ofdeviations along separate axes (e.g., up and left) may be denoted byshading, for example a top left corner (to include a left-hand edge anda top edge) of graphical display 110, and so on.

In some embodiments, the concept of shading may be extended to displaydevice 108, where at least one portion of display device 108 is shadedbased on a deviation of the user's current posture from thepredetermined posture. The methods and techniques described above forshading graphical display 110 may also be used on display device 108. Ina particular embodiment, one or more edges associated with displaydevice 108 may be shaded depending on the direction of the deviation.For example, if the deviation is to the right of the user, a right-handedge or a right-hand portion of display device 108 may be shaded toalert the user of the deviation and the associated direction. Or, if thedeviation is to the left and down, a lower left-hand corner (to includea bottom edge and a left-hand edge) of display device 108 may be shaded.

In some embodiments, posture analysis system 100 may remove thegraphical image such as graphical image 110 or graphical image 808 fromdisplay device 108 after a certain time interval once the user assumesand maintains the predetermined posture. In some embodiments, processingsystem 102 may process visual data at a lower frame rate once the userhas assumed the predetermined posture with graphical image 110 removedfrom display device 108. This is done to reduce any computing resourceutilization by posture analysis system 100 on computing system 102 whenthe user maintains the predetermined posture. When the user deviatesfrom the predetermined posture and needs interactive feedback,processing system 102 may process visual data at a higher frame rate toprovide substantially real time updates to the user via graphical image802.

In some embodiments, the user's shoulders will be indicated both withthe predetermined posture and the current user posture with a box and acolorful icon which should be in the box, or by using any othergraphical, textual or audible feedback method. In some instances, theuser's head might move forward but the shoulders stay in place. Thismotion implies that the user is craning their head forward. Or, theuser's shoulders may be moving up left and/or right. This posturesuggests that the user may be shrugging. As another example, the user'sshoulders may be rendered as falling down left and/or right. This isassociated with the possibility of a deflating-kind of posture. In someembodiments, user messages presented in response to, for example, theuser shrugging may be similar to “You are shrugging left” or “You areshrugging right,” depending on which direction the user is shrugging.The corresponding visual depiction could include a close-up of ashoulder, neck, part of head that shows a shoulder out of place and adownward arrow suggesting that the user lower their shoulder.

FIGS. 7 and 8 characterize the three-dimensional operational ability ofposture analysis system 100. The individual scenarios indicated in FIGS.7 and 8 are example representations provided for purposes ofexplanation. In some embodiments, a user posture may be a combination ofthe scenarios depicted in FIGS. 7 and 8, which would constitute a fullthree-dimensional representation. Being able to dynamically suggestposture corrections allows posture analysis system 100 to detect andsuggest corrections for unhealthy postures such as slouching, shrugging,craning of the neck, and so on by a user. Including the position of theuser's shoulders provides increased capability to the system in someconditions such as craning of the neck, while also allowing postureanalysis system 100 to characterize the position of the user's head inrelation to the user's shoulders. In particular embodiments, theposition of the user's face is sufficient to characterize the user'scurrent posture and provide associated posture correction feedback.

In some embodiments, posture analysis system 100 can also provideadditional visual cues that help the user correct their posture. Forexample, suppose the deviation in the current posture of the user fromthe predetermined posture is to the left. Posture analysis system mayshade a portion of the left-hand side of graphic element 114 to providea visual cue to the user to shift to the right. Sections of graphicelement 114 may be shaded in a similar fashion, respectivelycorresponding to deviations in the user's posture from the predeterminedposture.

In some embodiments, posture analysis system 100 can also providesuggestions other than posture corrections. These suggestions may beaimed at, for example, reducing user fatigue. For instance, postureanalysis system 100 may detect that a user may have assumed thepredetermined posture without moving for a particular period of time.Since assuming a particular posture for long periods of time withoutmoving is also unhealthy, posture analysis system 100 can provide a userwith prompts, suggesting that they get up, take a break, shift theirposition, or stretch.

The system may also include blink detection. A facial detectionalgorithm may include an ability to detect and monitor a user's eyes. Ifposture analysis system 100 determines that the user has not blinked fora period of time, posture analysis system 100 can prompt the useraccordingly, thus aiming to reduce eye fatigue.

To reduce resource utilization by posture analysis system 100 oncomputing system 102, posture analysis system 100 may use a relativelyhigh frame rate of 30 frames per second of visual data when a deviationis detected in the user's current posture as compared to thepredetermined posture, such as during the time when graphical image 110is being displayed to the user along with the associated feedback. Whenthe user has assumed the predetermined posture after correcting for thedeviation, graphical image 110 may be removed and the processing framerate may be reduced.

In some embodiments posture analysis system 100 may include a lag ofapproximately ⅓ seconds, or 320-370 milliseconds to account for a userswaying back and forth about a mean position that is the predeterminedposture, a process considered acceptable and encouraged for maintaininghealthy posture. Without the built-in lag, posture analysis system 100may exhibit an overly sensitive response to user inputs. In alternateembodiments, any value may be used as a time lag in posture analysissystem 100.

In some embodiments, a user may be able to configure the linerepresentation associated with graphical image 110, and may have accessto customized avatars to allow them to customize the graphical image totheir taste. In some embodiments, the colors associated with graphicalimage 110 may be automatically configured by posture analysis system 100in accordance with importance. For example, an initial deviation in theuser's posture may be rendered on graphical image 110 in subtle colors(for example, green). However, these colors may be replaced by stronger,more saturated colors (e.g., red or orange) if the user does not payattention to the posture correction alerts for a certain period of time.As another example, yellow may be used to denote an initial deviation inthe user's posture relative to the predetermined posture, orange may beused for a longer time period associated with the deviation, while aneven longer time period associated with the deviation may be rendered inred. Also, blue may be used to alert the user if they are frozen in aparticular posture for greater than, for example, 30 seconds. In someembodiments, vibrant primary colors may be used to capture the user'sattention from the corner of their eye while they are focused on anothertask on, for example, display system 108. These colors could be appliedto, for example, the text message, the line rendition of thepredetermined posture, and the ellipse corresponding to the rendition ofthe user's face. In some embodiments, the time period associated withchanging the colors displayed on graphical image 110 due to lack of userresponse may range from 2 seconds to 30 minutes. In particularembodiments, the size of graphical image 110 in relation to displaysystem 106 may also be increased in accordance with a user notresponding to alerts for a certain time period.

In some embodiments, visual data provided by imaging device 106 may notbe of acceptable quality if light levels are low. This limitation maybe, for example, due to the small apertures associated with the lensincorporated into imaging device 106. To work around this limitation, acolor image from imaging device 106 may be converted into a grayscaleimage that is processed in a histogram equalization algorithm thatbalances the darkest and lightest regions of the image to provide abetter foundation for cascade classifier face detection algorithms. Thegrayscale image is again processed in a gamma correction algorithmcalled gamma compression that lightens the image. In some embodiments,the grayscale image is processed using the Contrast Limited AdaptiveHistogram Equalization algorithm that specializes in producing photorealistic image and sensor data in low lighting conditions resulting inan image that can be successfully analyzed by face detection algorithmseven in low light conditions. In some embodiments, any combination ofimage processing algorithms may be used depending on ambient lightlevels. In particular embodiments, image processing algorithms may beapplied individually to the different color channels (e.g., red, greenand blue) of a color image to perform operations such as facialrecognition or face detection.

Advice may also be given to the user to take a walk or other correctiveaction if improper posture occurs over a period of time. For example,the system may determine how long a person is sitting in front of acomputer and in which posture. So a reminder to stretch, walk, drink,move, etc. can be triggered not just based on a timer (e.g., 35 to 45minutes); the reminder can be triggered smartly because the system knowshow long the user was sitting and the quality of their posture. If theuser sits in a bad posture (i.e., they have a hard time holding theirown ideal posture) the system can remind the user after a shorter periodof time, such as 15 to 20 minutes. On the other hand, if the usermaintains their ideal posture, the reminder time can be longer (e.g., 35to 60 minutes). Also if the user consistently has a hard time sitting intheir chosen good posture, the system can ask them to get help withtheir work station setup or posture they believe to be optimal. Or,posture analysis system 100 may recognize that the user might havechanged the position of the computer (camera) or chair, or that thelighting situation has changed so significantly that the software needsto render a new good posture (predetermined posture), necessitating aposture recalibration.

In some embodiments, a user could choose to set posture analysis system100 to give feedback strictly so that the slightest deviation from goodposture results in feedback to the user. Conversely, the user canexplicitly set the system to only provide feedback when the user is farout of good posture, a so-called looser interpretation of good posture.In addition, the user could explicitly choose to receive frequentfeedback about their posture or explicitly choose to receive infrequentfeedback such as at the end of the session, end of the day, end of theweek or some other time frame. The system could also determine howstrict/loose to set the posture calculation and how frequent/infrequentto give the user feedback from a variety of implicit user factors suchas how long the user has used the posture tracking system, how often theuser dismisses or ignores feedback, or how fast the user responds tofeedback and the like.

FIG. 9 is a schematic diagram depicting an embodiment of a sequence offrames 900 to determine a user's posture. In some embodiments, a frame902 associated with visual data from imaging device 106 includes aprofile 904 of a user that is determined as a foreground mask based onthe area of any contours of foreground segments in frame 902. In someembodiments, profile 904 is determined as a largest blob from theforeground, where a user's position is illustrated in a general outlineof the person's body and is referred to as a “blob.”

In some embodiments, a frame 906 evolves from frame 902, where frame 906is generated by calculating a convex hull 908 over a largest blobcontour 910 associated with the profile 904. A convex hull is defined asa simplified shape of a primary object that is being tracked, in thiscase the user's posture. In some embodiments, convex hull 908 moves inaccordance with any shift in a user's posture. Based on a position of aconvex hull associated with a predetermined posture, the instantaneousposition of convex hull 908 relative to the position of the convex hullassociated with the predetermined posture provides a measure of theassociated shift in the user's posture from the predetermined posture.This forms the basis for a posture tracking algorithm. In someembodiments, the largest blob contour may be identical to profile 904.

In some embodiments, additional processing on the visual data associatedwith frame 906 gives a frame 912 that includes a user profile 914 (whichmay be identical to profile 904) as well as a detected user face 916. Insome embodiments, detected user face 916 may be determined using OpenCVor other algorithms such as frontal face and eye cascades.

FIG. 10 is a schematic diagram depicting an embodiment of a workflowthat illustrates the operation of posture analysis system 100. During asystem initialization process 1002 (denoted by input in FIG. 10),posture analysis system 100 initializes receiving data from imagingdevice 106. Input from imaging device 106 may be represented by a frame1004 that may include visual information about a user in the field ofview of imaging device 106. In some embodiments, a backgroundinitialization step 1006 characterizes a background model 1008 in theabsence of the user (i.e., when the user is not in the field of view ofimaging device 106). This activity allows posture analysis system 100 toappropriately determine the presence of the user as a foreground elementto accomplish posture tracking. In some embodiments, a machine learningclassifier may be used to create a background model. In otherembodiments, background initialization 1006 may be skipped altogether.

A user initialization step 1010 characterizes a predetermined postureassociated with a user. In some embodiments, the user positionsthemselves in the field of view of imaging device 106 to performinitialization step 1010. In some embodiments, initialization step 1010includes receiving a frame 1014 containing an image of the user, usingimage processing techniques (such as convex hull and blob analysis) togenerate a frame 1016 containing a convex hull associated with theuser's image profile, using face detection and facial detectiontechniques to confirm a presence of the user's face in a generated frame1018. Additionally, the system generates a frame 1020 containing anoutline of the user's profile, finally generating a frame 1012 thatconfirms the outline of the user's profile and storing data associatedwith frame 1012 as the predetermined posture. In some embodiments, thepredetermined posture associated with user initialization step 1010 maybe determined via professional input such as input from a therapist,ergonomics professional, a coach, in person, or via the internet in avideo chat, general information from the Internet, and so on.

A posture tracking step 1022 tracks the user's posture in substantiallyreal time. In some embodiments, the user positions themselves in thefield of view of imaging device 106 to perform posture tracking step1022. In some embodiments, posture tracking step 1022 includes receivinga frame 1026 containing an image of the user, using image processingtechniques (such as convex hull and blob analysis) to generate a frame1028 containing a convex hull associated with the user's image profile.In some embodiments, posture tracking step 1022 uses face detection andfacial detection techniques to confirm a presence of the user's face ina generated frame 1030, generating a frame 1030 containing an outline ofthe user's profile relative to the predetermined posture. Next, posturetracking step 1022 generates a frame 1032 that confirms the outline ofthe user profile. Finally, posture tracking step 1022 generates a frame1024 that confirms the outline of the user's profile and current posturerelative to the predetermined posture. In some embodiments, posturetracking step 1022 as implemented by posture analysis system 100 may becapable of re-initializing posture tracking of a user even when the userreturns after having exited from the field of view of imaging device106. In some embodiments, posture tracking step 1022 as implemented byposture analysis system 100 can detect a physical movement or otherimpairment in imaging device 106, and can autonomously recognize theneed to redo the background learning and posture calibration steps,without the user needing to take any actions.

In some embodiments, background initialization step 1006, userinitialization step 1010, and posture tracking step 1022 may be carriedout in any order, not necessarily in the sequence presented in FIG. 10.In some embodiments, posture analysis system 100 may be configured todetect when the user has left the field of view of the camera based on,for example, information stored from background initialization step1006.

In some embodiments, a user interface presented by posture analysissystem 100 may include buttons and a visual display. The visual displaymay include a live-video display of the scene captured by the camera. Ifthe user is recognized in the scene, a mask is displayed around theuser's boundary. The mask is a region of pixels outside the boundary ofthe user's body as detected by posture analysis system 100, and theregion may be blurred such that the background scene appears opaque. Inoperation, the user's objective is to keep their body within the clearboundary of pixels. Thus, the user may observe a visual image of theircurrent posture in the form of a boundary in which they may position theuser's body. When they move out of the boundary, they can be made awareof this as their actual position is shown outside the boundary shown inthe image.

The ideal posture can also be indicated by a simple outline or bycreating an avatar of the real person inside of the person's outline.The user may then change or otherwise correct their position so that theimage of their actual position coincides with the image projected in thevisual boundary shown to them. When an optimum position is achieved, thevisual representation may indicate that the user is in the properposition, which may change if a user moves. For example, the image mayappear bright green when a user is in a preferred position, and maychange to red if the user poses outside the boundary of the preferredposition. Audible alerts may also be incorporated, alerting the userthat their posture is outside the boundaries. The appearance or audiblesound of such an alert may be varied depending on the preferred userexperience, avoiding annoying alerts that might distract or distress theuser.

In some embodiments, an output step 1034 presents posture trackingresults to a user via a display output 1036. In some embodiments,display output 1036 is identical to graphical image 110. In someembodiments, an image of the user and an outline of the predeterminedposture may be presented. In other embodiments, the image of the usermay be replaced with a graphic element such as an ellipse. Depending onthe deviation of the user's current posture from the predeterminedposture, display output 1036 may show different colors. For example, ifthe user has assumed the predetermined posture display output 1036 mayuse a green color for rendering certain elements. If the user deviatesfrom the predetermined posture, display output 1036 may use other colorssuch as yellow or red, depending on the severity of the deviation.

Posture analysis system 100 can be used to correct posture at a computerdesk or working in proximity to any type of display device. Postureanalysis system 100 can also be used to guide users in full bodypostures and movements, if the visual data from imaging device 106 isinterpreted and sent to (3D)-video-glasses, helmet-mounted displays, ora fixed monitor, or if the positioning feedback is verbalized and playedas audio feedback to the user. Posture analysis system 100 is alsouseful in situations such as yoga, physical therapy, exercise,stretching, or any other learning of body movement where the sequenceand feedback would be at least similar to the posture feedback discussedherein. Other applications include realizations in airplanes, buses,trains, and cars, where the immediate background is fairly static. Inthese applications, the background may be less relevant with regards toprocessing functions, and the focus is primarily on the user in front ofimaging device 106.

FIG. 11A represents a flow diagram depicting an embodiment of a method1100 to track and correct a user's posture based on image recognition.At 1102 the method receives visual data associated with a user's face,shoulders and chest. In some embodiments, the visual data may bereceived by processing system 102 from imaging device 106. At 1104, themethod extracts information about a largest blob in a frame associatedwith the visual data. In some embodiments, the largest blob isassociated with a foreground rendition of the user in the frame, toinclude the user's head, shoulders and chest. At 1106, the methoddetermines a convex hull associated with the largest blob, where aconvex hull is defined as the smallest convex set that contains all thepoints of the largest blob. Next, at 1108, the method determines whetherthe convex hull includes the user's face, using techniques such asfacial detection. If the convex hull does not include the user's facethen the method goes to B and returns back to 1102, where the systemreattempts to detect the presence of the user's face in the visual data.On the other hand if, at 1108, the convex hull includes the user's facethen the method goes to 1110, where the method determines a position ofthe user's shoulders and a position of the user's chest in the framebased on the location of the user's face and the position of the convexhull. Next, at 1112, the method determines a posture of the user basedon relative positions of the user's face, the user's shoulders and theuser's chest. The method then goes to A, with a continued description inFIG. 11B.

FIG. 11B is a continued description of method 1100. Starting at A, themethod goes to 1114, where the method tracks the posture of the userwith respect to time using the posture tracking methods describedherein. At 1116 the method checks to see whether the posture of the useris substantially different from a predetermined posture. If the postureof the user is not substantially different from the predeterminedposture then the method goes to B, and returns back to 1102. On theother hand if, at 1116, the posture of the user is substantiallydifferent from the predetermined posture then the method goes to 1118,where the method alerts the user if the posture of the user issubstantially different from the predetermined reference posture usingat least one of an audible alert or a visual alert.

FIG. 12A represents a flow diagram depicting an embodiment of a method1200 to characterize a user's posture. At 1202, the method receivesvisual data associated with a user's head, shoulders and chest. In someembodiments, the visual data may be received by processing system 102from imaging device 106. At 1204, the method determines a largest blobin a frame associated with the visual data. Next, at 1206, the methoddetermines a convex hull of a contour associated with a largest blob.The process of determining the largest blob and associated convex hullis performed using methods described earlier.

At 1208, the method detects the user's face in the convex hull using,for example, OpenCV or frontal face and eye cascades. Next, at 1210, themethod determines the presence of the user's face in the frame. At 1212,the method determines a position of the center of the user's head basedon the position of the user's face, while at 1214 the method determinesa position of the bottom of the user's head. The method then goes to A,with a continued description provided subsequently.

FIG. 12B is a continued description of the method 1200. Starting at A,the method goes to 1216, where the method determines a position of theuser's left shoulder based on the convex hull. Next, at 1218, the methoddetermines a position of the user's right shoulder based on the convexhull. In some embodiments, the position of the user's left shoulder andthe position of the user's right shoulder are determined based on usermotion, with the user swaying side-to-side. Using image recognition orcomputer vision algorithms, the user's left shoulder is located byfinding the center of the user's head and the bottom of the user's headwithin the user motion region and then splitting the user motion regionin half wherein the left side portion of the user motion region locatedbeneath the head and to the left of the center of the head is consideredthe left shoulder. Though this region will contain non-shoulder data dueto the user movement, it will still be able to use this shoulder data tofind the shoulder in the posture tracking step. Similarly, the user'sright shoulder is located by finding the center of the user's head andthe bottom of the user's head within the user motion region and thensplitting the user motion region in half wherein the right side portionof the user motion region located beneath the head and to the right ofthe center of the head is considered the right shoulder. Though thisregion will contain non-shoulder data due to the user movement, it willstill be able to use this shoulder data to find the shoulder in theposture tracking step.

Finally, at 1220, the method characterizes the user's posture using theposition of the user's face, the position of the user's left shoulder,and the position of the user's right shoulder.

FIG. 13 is a schematic diagram illustrating an embodiment of anapplication 1300 of posture tracking system 100 in an airplane cabin1302. In some embodiments, a user 1304 may install and run posturetracking system 100 on a mobile device 1306 and place (or mount) themobile device on the back of the seat in front of the user as shown inFIG. 13. In some embodiments, a user mobile device such as a mobilephone or a tablet may be mounted on the back of the seat in front of theuser using plastic clips. In some embodiments, a user may place theirlaptop computer with integrated webcam on top of a tray table in frontof the user. User 1304 may initialize posture tracking system 100 with apredetermined posture using the built-in camera of mobile device 1306.Then, posture tracking system 100 can track the posture of user 1304 forthe period of the journey or for a time period specified by the user.

FIG. 14 is a schematic diagram illustrating an embodiment of anotherapplication 1400 of posture tracking system 100 in an automobile orother vehicle to provide a posture tracking feature for the driver ofthe automobile. FIG. 14 shows a vehicle instrument cluster 1402, withposture tracking system 100 providing posture feedback to a driver ofthe vehicle on a vehicle display 1404. Some embodiments may use amounted camera on the vehicle dashboard in front of the user to monitorthe user's posture, while some embodiments may use a camera that isinstalled in the vehicle dashboard to monitor the user's posture.

Other embodiments of posture analysis system 100 may include animplementation in an operating room where an imaging device is focusedon the doctor or other personnel and is coupled to a computing systemand associated display device to provide posture feedback to the doctoror other personnel. This system would provide feedback for situationssuch as when a user's shoulders are not square to the floor for extendedtime (shrugging). Also a user having their arms in fixed positions forlong periods of time (either totally straight or elbows bent) may beaddressed by posture analysis system 100. General slouching (downwardmovement of body from an initial good posture position) may also beaccounted for. The imaging device facing the doctor could be a laptop orwebcam or phone/tablet placed on a moveable arm tray positioned abovethe patient in the line of sight of the doctor, or anywhere that isdoctor facing as long as the doctor's face is within the field of viewof the imaging device.

Although the present disclosure is described in terms of certain exampleembodiments, other embodiments will be apparent to those of ordinaryskill in the art, given the benefit of this disclosure, includingembodiments that do not provide all of the benefits and features setforth herein, which are also within the scope of this disclosure. It isto be understood that other embodiments may be utilized, withoutdeparting from the scope of the present disclosure.

1. A method comprising: identifying, by a computing system, a deviationof a user's posture from a predetermined posture based on visual dataassociated with the user; informing the user, via a graphical imagedisplayed on a display device, of the deviation; and providinginstructions, via the graphical image displayed on the display device,for correcting the deviation.
 2. The method of claim 1, wherein the useris informed of the deviation in substantially real time.
 3. The methodof claim 1, wherein informing the user of the deviation includesdisplaying an image depicting the predetermined posture andsimultaneously displaying a current representation of the user'sposture.
 4. The method of claim 3, wherein the displayed image isupdated in substantially real time.
 5. The method of claim 1, whereinthe graphical image is displayed responsive to the deviation being abovea predetermined threshold.
 6. The method of claim 1, wherein identifyingthe deviation of the user's posture includes performing facial detectionon the received visual data.
 7. The method of claim 6, furthercomprising determining a position of the user's shoulders in the visualdata, wherein the position of the user's shoulders is used to determinethe deviation.
 8. The method of claim 1, further comprising:determining, based on visual data associated with the user, whether theuser has corrected their posture; and removing the graphical imageresponsive to determining that the user has corrected their posture. 9.The method of claim 1, wherein the instructions include at least one ofa textual message, a graphical symbol, audio feedback, or shading aportion of the graphical image based on the deviation.
 10. The method ofclaim 1, further comprising color-coding the instructions based on atime duration associated with the deviation.
 11. The method of claim 1,further comprising providing a congratulatory message to the user whenthe deviation has been corrected.
 12. A method comprising: receiving, bya computing system, visual data associated with a user; determining auser's current posture in three dimensions based on the visual data,wherein determining the user's current posture includes: determining theuser's posture along a first axis based on a size of the user's face;determining the user's posture along a second axis based on a firstposition of the user's face along the second axis; and determining theuser's posture along a third axis based on a second position of theuser's face along the third axis; and tracking the user's currentposture in three dimensions.
 13. The method of claim 12, furthercomprising detecting, based on the visual data, a position of the user'sleft shoulder and a position of the user's right shoulder.
 14. Themethod of claim 13, wherein determining the user's current postureincludes the position of the user's left shoulder and the position ofthe user's right shoulder.
 15. The method of claim 12, wherein trackingthe user's current posture includes: tracking the user's current posturerelative to a predetermined posture; and identifying a deviation of theuser's current posture from the predetermined posture.
 16. The method ofclaim 12, further comprising filtering, based on processing performed onthe visual data by at least one of averaging functions, smoothingfunctions, or probabilistic functions, any erroneous jumps in trackingresults caused due to measurement errors associated with the computingsystem.
 17. The method of claim 12, further comprising detecting, usingimage recognition, the user's eyes, nose and lips based on the visualdata to enhance characterization of the user's posture.
 18. A methodcomprising: identifying, by a computing system, a deviation of a user'sposture from a predetermined posture based on visual data received bythe computing system; and associating the deviation of the user'sposture with a specific task being performed by the user.
 19. The methodof claim 18, further comprising recording a history corresponding to theassociation of the deviation of the user's posture with the specifictask.
 20. The method of claim 18, wherein the specific task is anapplication executing on the computing system.